Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Informing bowing and violin learning using movement analysis and machine learning

Violin performance is characterized by an intimate connection between the player and her instrument that allows her a continuous control of sound through a sophisticated bowing technique. A great importance in violin pedagogy is, then, given to techniques of the right hand, responsible of most of the sound produced. This study analyses the bowing trajectory in three different classical violin exercises from audio and motion capture recordings to classify, using machine learning techniques, the different kinds of bowing techniques used. Our results show that a clustering algorithm is able to appropriately group together the different shapes produced by the bow trajectories

MIE 2017: 1st international workshop on multimodal interaction for education (workshop summary)

The International Workshop on Multimodal Interaction for Education aims at investigating how multimodal interactive systems, firmly grounded on psychophysical, psychological, and pedagogical bases, can be designed, developed, and exploited for enhancing teaching and learning processes in different learning environments, with a special focus on children in the classroom. Whilst the usage of multisensory technologies in the education area is rapidly expanding, the need for solid scientific bases, design guidelines, and appropriate procedures for evaluation is emerging. Moreover, the introduction of multimodal interactive systems in the learning environment needs to develop at the same time suitable pedagogical paradigms. This workshop aims at bringing together researchers and practitioners from different disciplines, including pedagogy, psychology, psychophysics, and computer science - with a particular focus on human-computer interaction, affective computing, and social signal processing - to discuss such challenges under a multidisciplinary perspective. The workshop is partially supported by the EU-H2020-ICT Project weDRAW (http://www.wedraw.eu)

Celiac disease, rare symptoms, autoimmune patology

We report a case of a 42-years-old woman with constipation, anemia and recurrent itch. After several investigations, celiac disease was diagnosed and a treatment with a gluten-free diet was applied with beneficial effects. Recognizing celiac disease can be difficult because some of its symptoms are similar to those of other diseases. In fact, sometimes it is confused with irritable bowel syndrome or iron-deficiency anemia or intestinal infections: as a result, celiac disease is commonly underdiagnosed or misdiagnosed. This case report is described to address the physician to a correct diagnosis of celiac disease

Automatic Detection of Reflective Thinking in Mathematical Problem Solving based on Unconstrained Bodily Exploration

For technology (like serious games) that aims to deliver interactive learning, it is important to address relevant mental experiences such as reflective thinking during problem solving. To facilitate research in this direction, we present the weDraw-1 Movement Dataset of body movement sensor data and reflective thinking labels for 26 children solving mathematical problems in unconstrained settings where the body (full or parts) was required to explore these problems. Further, we provide qualitative analysis of behaviours that observers used in identifying reflective thinking moments in these sessions. The body movement cues from our compilation informed features that lead to average F1 score of 0.73 for binary classification of problem-solving episodes by reflective thinking based on Long Short-Term Memory neural networks. We further obtained 0.79 average F1 score for end-to-end classification, i.e. based on raw sensor data. Finally, the algorithms resulted in 0.64 average F1 score for subsegments of these episodes as short as 4 seconds. Overall, our results show the possibility of detecting reflective thinking moments from body movement behaviours of a child exploring mathematical concepts bodily, such as within serious game play

Early Occupational Therapy Intervention: Patients’ Occupational Needs

The occupational therapy management involves the assessment of the individual’s specific needs. This kind of assessment facilitates the therapeutic relationship and boosts the person’s motivation, as he or she feels valued and heard. Early-stage collection of information about meaningful activities for the individual helps them project themselves outside the context of illness. Collecting occupational need at an early stage, permits “Engagement”, which means participating in activities even without actually doing them. An occupational therapy model called “Personal Environment Occupation Model” suggests that already at an early stage we should make the environment and occupations meaningful to the person in order to maximise the patient’s performance. An observational study on stroke patients shows how people have personal occupational needs beyond simple self-care, including productive life and leisure time, already in the subacute phase. A further study is underway to demonstrate the effectiveness of early occupational therapy intervention, including complex patients regardless of diagnosis and taking into account their need for care and disability in order to promote their participation and maximise their autonomy

Automated analysis of postural and movement qualities of violin players

Learning to play a music instrument is a complex task, requiring continuous practice and the development of sophisticated motor control techniques. The traditional model of music learning is based on a master-apprentice relationship, leading often to a solitary learning process, in which the time spent with the teacher is usually limited to weekly lessons and a long period of self-study is needed. Moreover, a large amount of time passes from the teacher's feedback and the student's proprioceptive perception while studying, requiring a big effort in developing an efficient and healthy technique. In this paper, we present our recent developments concerning an assistive and adaptive technology to help violin students overcoming all these difficulties, and developing their technique and repertoire properly and safely. In particular, we focus on the multimodal corpus of violin performances which was collected for the purpose, and on the analysis of such data to compute postural and gestural features characterizing the performance under a biomechanical perspective and in terms of movement quality. Analysis is expected to provide students with feedback for reaching a physically accurate performance, maximizing efficiency and minimizing injuries

Designing a Multisensory Social Serious-Game for Primary School Mathematics Learning

Psychophysical and developmental psychology evidence shows that children have a preferential sensory channel to learn specific concepts, highlighting the need of a multisensory approach to education. Despite such a potential for a positive impact, in education technology research multisensory learning has been often penalised and limited by design choices. In this work, we present the requirements and the design of an implementable prototype of serious-game aimed at teaching children a precise mathematical concept: Angles. The game is based on the interaction between different sensory modalities, in particular movement, vision, and sound

Enhancing children understanding of mathematics with multisensory technology

Recent results from psychophysics and developmental psychology show that children do not integrate and use the same sensory modalities in the same way, but they rather have a preferential sensory channel to learn specific concepts. In schools, however, the visual channel is often the only one exploited for teaching, whereas the other channels are left to a marginal role. The weDRAW project, an ongoing European two-years project, explores the chance to create and evaluate new methodologies to teaching, grounded on multisensory technologies. In particular, the project targets a deeper understanding of arithmetic and geometry. A major novelty of such a new technology is that it is based on the renewed understanding of what communication between senses is and how it occurs during child development, that is, that specific sensory systems have specific roles for learning specific concepts. Neuroscientific prospectives suggest that it is possible to develop new teaching/learning channels, personalized for each student based on the child's sensory skills. This practice work consists of a demo simulating a set-up in a primary school, where children with any level of previous mathematical knowledge can try the technologies themselves. The demo includes full-body activities to "play" and "draw" arithmetical and geometrical concepts, such as playing a fraction using body rhythms and drawing shapes and symmetries

How unitizing affects annotation of cohesion

This paper investigates how unitizing affects external observers' annotation of group cohesion. We compared unitizing techniques belonging to these categories: interval coding, continuous coding, and a technique inspired by a cognitive theory on event perception. We applied such techniques for sampling coding units from a set of recordings of social interactions rich in behaviors related to cohesion. Then, we compared the cohesion scores the observers assigned to each coding unit. Results show that the three techniques can lead to suitable ratings and that the technique inspired to cognitive theories leads to scores reflecting variability in cohesion better than the other ones